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WWC Review of the Report “A Written Language Intervention for 
At-Risk Second Grade Students: A Randomized Controlled Trial 
of the Process Assessment of the Learner Lesson Plans in a 
Tier 2 Response-to-lntervention (Rtl) Model” 1 

The findings from this review do not reflect the full body of research evidence 
on Process Assessment of the Learner. 


What is this study about? 

The study examined the effects of Process Assess- 
ment of the Learner {PAL), a writing expression cur- 
riculum. The program was tested with second-grade 
students in a suburban-rural school district in the 
southeastern United States. Three sections of PAL 
lessons were implemented in the district as a small- 
group curriculum supplement— Talking Letters, Spell- 
ing, and Handwriting and Composition. 

Study authors randomly assigned two cohorts of 
at-risk students in seven schools to either an inter- 
vention group that received PAL (68 students) or a 
comparison group that did not receive PAL (70 stu- 
dents). 2 At-risk students were defined as students 
who scored at or below the 25th percentile for their 
grade on the Wechsler Individual Achievement Test- 

11 (WIAT-II) Written Expression Subtest. 

Students in the intervention group received PAL 
lessons taught by research associates and gradu- 
ate students during second grade in small groups of 
three to six students in addition to their usual written 
language instruction. The small groups received a 
series of 24 sessions lasting 25 minutes each over 

12 weeks. Students in the comparison group contin- 
ued to receive the usual written language instruction 
in second grade, a statewide standard curriculum in 
which writing skills were embedded in daily class- 
room activities with little direct instruction. The 


standard curriculum included ongoing development 
of alphabetic principles; using vocabulary in written 
communication; composing written sentences; plan- 
ning and composing narrative texts; and capitaliza- 
tion, punctuation, syntax, and grammar. 

The researchers examined the effects of PAL by 
comparing the performance of 66 students in the 
intervention group and 63 students in the com- 
parison group (after attrition) on the WIAT-II Written 
Expression Subtest in the fall of third grade. 3 

In addition to the analysis that compared at-risk stu- 
dents in the intervention and comparison conditions, 
the authors also compared the intervention group 
(at-risk students who received PAL) to a sample of 
typical students (defined as scoring higher than the 
25th percentile on the writing assessment) who were 
not included in the experiment and therefore did not 
receive PAL. 
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Features of Process Assessment 
of the Learner (PAL) 


PAL is a writing expression curriculum that is 
available in three tiers of instructional intensity 
(classroom, small group, and individual student). 
Writing expression encompasses the ability to 
communicate and express thoughts in written form. 
The study delivered Reading and Writing Lesson 
Sets 4 and 7 over the course of 12 weeks, which 
included the following sections: 

• In the subword level Talking Letters section, 
students used cards with visual cues of phonemes 
that they named in rapid succession. 

• The word level Spelling section focused on eight 
words per instructional session, asking students to 
use their phoneme strategies to spell words orally 
and in written form. 

• For the text level Handwriting and Composition 
section, students responded in writing to a 
prompt, using a series of six high-frequency words 
per session. 

• Some lessons included handwriting instruction, in 
which students looked at a model and practiced 
the correct construction of each letter. 

This study focuses on the effect of the small group 
implementation of PAL. 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards without reservations 

The analysis of the impact of PAL on the 
achievement of at-risk students is a well-executed 
randomized controlled trial with low attrition. 
However, the authors’ non-experimental comparison 
of the performance of at-risk students receiving PAL 
to the performance of typical students did not meet 
WWC evidence standards because the groups were 
not equivalent at baseline. 4 


What did the study find? 

The study authors found that the average written 
expression skills of the PAL intervention group were 
higher than those of the comparison group at the 
beginning of third grade, based on an analysis of 
growth trajectories. However, the WWC did not con- 
firm that the observed effect of the PAL intervention 
on growth in written expression skills was statisti- 
cally significant . 5 
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Appendix A: Study details 

Hooper, S. R., Costa, L. C., McBee, M., Anderson, K. L., Yerby, D. C., Childress, A., & Knuth, S. B. (2013). 
A written language intervention for at-risk second grade students: A randomized controlled trial 
of the process assessment of the learner lesson plans in a tier 2 response-to-intervention (Rtl) 
model. Annals of Dyslexia, 66(1), 44-64. 


Setting 

The study was conducted in a single suburban-rural school district in the southeastern United States. 

Study sample 

First-grade students in two cohorts in seven schools were assessed on written expression 
using the WIAT-II Written Expression Subtest. Students who scored at or below the 25th per- 
centile on the WIAT-II were identified as at-risk students. Students who scored above the 25th 
percentile on the WIAT-II were labeled typical students. 

The at-risk students were randomly assigned to either the intervention or the comparison 
group. The analysis sample of at-risk students included 66 students in the intervention group 
and 63 students in the comparison group. A random sample of typical students (students 
scoring higher than the 25th percentile on the writing assessment), balanced across schools 
and classrooms, was used for a contrast (with at-risk students who received PAL) that did 
not meet WWC evidence standards because these students were not randomly assigned to 
experimental conditions and were not shown to be equivalent at baseline. 

Intervention 

group 

Students in the intervention group in both cohorts received supplemental small group instruc- 
tion in PAL during second grade in addition to their usual written language instruction. The 
small groups consisted of three to six students and one project-based interventionist who met 
for 24 sessions of 25 minutes each for 12 weeks. The PAL lessons focused on recognizing 
phonemes, spelling, composition, and handwriting instruction. Lessons were designed to be 
35 to 40 minutes long, but were modified to meet the available time provided by the school 
system while still retaining the integrity of the original plans. 

Comparison 

group 

Students in the comparison group received the usual written language instruction during sec- 
ond grade following a statewide standard course of study. This curriculum included ongoing 
development of alphabetic principles; using vocabulary in written communication; composing 
written sentences; planning and composing narrative texts; and capitalization, punctuation, 
syntax, and grammar. These skills were embedded in daily classroom activities with little direct 
instruction for written expression. 

Outcomes and 
measurement 

The study authors examined scores from the WIAT-II Written Expression Subtest. Pretests 
were administered in the fall of second grade, and posttests were administered in the fall 
of third grade, following the end of the intervention at the end of second grade. For a more 
detailed description of this outcome measure, see Appendix B. 

Support for 
implementation 

Research associates and graduate students in education and school psychology administered 
the intervention. They were trained on the PAL lesson plans and how to record the reliability 
of each component. They also completed a lesson checklist on compliance with each inter- 
vention component after each PAL session. The lead interventionist worked with the project 
interventionists by discussing lesson plans prior to implementation, conducting random obser- 
vations of sessions with follow-up, and holding weekly discussions. 
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Reason for This study was identified for review by the WWC because it was supported by a grant to The 
review Carolina Institute for Developmental Disabilities at the University of North Carolina School of 
Medicine (Principal Investigator: Stephen R. Hooper) from the National Center for Education 
Research (NCER) at the Institute of Education Sciences (IES). 
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Appendix B: Outcome measure for the written expression domain 


Written expression 


Wechsler Individual Achievement Test-ll This measure consists of a series of cognitive and psychoeducational measures drawn from different normed 
(WIAT-II) Written Expression Subtest and standardized assessments. At grades 1 and 2, the Written Expression Subtest consists of three tasks: 

timed alphabet writing, written word fluency, and sentence combining. At grade 3, the student is asked to write 
a paragraph in accordance with a specific writing prompt. The score analyzed is the raw score. Past reported 
inter-item reliability for this assessment ranged from 0.91-0.94. 
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Appendix C: Study findings for the written expression domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Written expression 

Wechsler Individual 

At-risk 

7 schools/ 

13.55 

12.40 

1.15 

0.22 

+9 

0.22 

Achievement Test-ll (WIAT-II) 

students 

129 students 

(5.04) 

(5.59) 






Written Expression Subtest 

Domain average for written expression 0.22 +9 Not 

statistically 

significant 

Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the average change expected for all students 
who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average student's percentile rank that can be expected if the student is given the intervention 

Study Notes: The WWC calculated the program group mean using a difference-in-differences approach (see the WWC Handbook) by adding the impact of the program (i.e., 
difference in mean gains between the intervention and comparison groups) to the unadjusted comparison group posttest means. Please see the WWC Procedures and Standards 
Handbook version 2.1 for more information. The authors’ analyses of the effects of PAL in the article examine rates of change in individual growth, and report model-based esti- 
mates of the differences across intervention and comparison groups at the beginning of third grade which are shown to be nonsignificant. The authors report that this difference 
is statistically significant when the model-based average at the beginning of third grade also subtracts out the difference in writing scores at the beginning of the treatment period 
(at the midway point of second grade). The WWC does not confirm this finding to be statistically significant, since this comparison effectively double counts the baseline difference 
across the groups (since the baseline scores are already included in the estimated growth trajectories). The p-values presented here were calculated by the WWC, based on infor- 
mation on sample sizes, group means, and standard deviations for the fall of second grade and fall of third grade assessments provided by the authors in a response to an email 
query. The study is characterized as having an indeterminate effect because the single effect is neither statistically significant nor substantively important. For more information, 
please refer to the WWC Standards and Procedures Handbook, version 2.1 , page 96. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the single study review protocol, version 2.0. 

2 The article indicates two different sample sizes for the intervention group (n = 68 and n = 69), but the authors confirmed that the 
total number of students randomly assigned to the intervention group was n = 68 in response to an email request for more information 
about the study. 

3 Our description of the timing of the assessments is based on the authors’ description in the text (“assessments were conducted 
in the fall of first, second, and third grade,” p. 50) and not on the numbering system the authors used to denote the timing of the 
assessments in their growth models. The authors describe the fall assessments taking place at time periods 1.5, 2.0, and 3.5, and the 
second grade winter and spring assessments for the intervention group taking place at time periods 2.5 and 3.0, respectively. For the 
purposes of this WWC report, we focus on the fall assessment in second grade as the pretest and the fall assessment in third grade 
as the outcome periods of interest. 

4 Because the two groups of students in this contrast were not randomly assigned to receive PAL , the WWC considers this analysis to 
be based on a quasi-experimental design that must demonstrate baseline equivalence to meet WWC evidence standards with reser- 
vations. The intervention and comparison groups in this contrast were not equivalent at baseline, so this portion of the study does not 
meet WWC evidence standards. 

5 The authors’ analyses of the effects of PAL in the article examine rates of change in individual growth, and report model-based esti- 
mates of the differences across intervention and comparison groups at the beginning of third grade which are shown to be nonsignifi- 
cant. The authors report that this difference is statistically significant when the model-based average at the beginning of third grade 
also subtracts out the difference in writing scores at the beginning of the treatment period (at the midway point of second grade). The 
WWC did not confirm this finding to be statistically significant, since this comparison effectively double counts the baseline difference 
across the groups (since the baseline scores are already included in the estimated growth trajectories). The WWC confirmed the non- 
significant differences in the groups at the beginning of third grade based on information obtained from the authors in an email query. 
The authors also present moderator analyses that examine whether impacts vary according to baseline assessments of cognitive 
variables and underlying latent classes, but the study does not provide sufficient information to assess whether these analyses meet 
WWC standards, and therefore, we do not include details about them in this review. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2014, February). 

WWC review of the report: A written language intervention for at-risk second grade students: A randomized 
controlled trial of the process assessment of the learner lesson plans in a tier 2 response-to-intervention (Rtl) 
model. Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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